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Abstract 

The  purpose  of  this  thesis  is  to  mathematically  remove  the  shadow  of  an 
individual  on  video.  The  removal  of  the  shadow  will  aid  in  the  rendering  of  higher 
quality  binary  silhouettes  than  previously  allowed.  These  silhouettes  will  allow 
researchers  studying  gait  recognition  to  work  with  silhouettes  unhindered  by  unrelated 
data. 

The  thesis  begins  with  the  analysis  of  videos  of  solid  colored  backgrounds.  A 
formulation  of  the  effect  of  shadow  on  specified  colors  will  aid  in  the  derivation  of  a 
hypothesis  test  to  remove  an  individual’s  shadow.  Video  of  an  individual  walking 
normally,  perpendicular  to  the  camera  will  be  utilized  to  test  the  algorithm. 

First,  the  algorithm  replaces  shaded  pixels,  pixel  values  determined  to  be 
shadows,  with  corresponding  pixels  of  an  average  background.  A  hypothesis  test  will  be 
employed  to  determine  if  a  pixel  value  is  a  shaded  pixel.  The  rejection  region  for  the 
hypothesis  test  will  be  determined  from  the  pixel  values  of  the  frames  containing  a 
subject. 

Once  the  shaded  pixels  are  replaced,  the  resulting  frames  will  then  be  run  through 
a  background  subtraction  algorithm  and  filtered,  resulting  in  a  series  of  binary  silhouettes. 
Researchers  can  then  utilize  the  series  of  binary  silhouettes  to  accomplish  a  gait 
recognition  algorithm. 
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STATISTICAL  REMOVAL  OF  SHADOW 


FOR  APPLICATIONS  TO  GAIT  RECOGNITION 


I.  Introduction 


Background 

All  individuals  possess  a  trait  that  is  identified  as  the  human  gait.  Several  aspects 
of  the  motion  of  natural  walking  can  be  decomposed  into  useful  mathematical  data. 
Examples  of  these  motions  are  the  length  of  a  stride,  the  angle  of  a  bended  knee,  the 
height  of  an  individual,  the  angle  between  two  legs  at  maximum  angle  apart,  et  cetera. 
This  information  is  extractable  in  lab  conditions  or  in  real  life  settings  with  the  proper 
equipment  and  set-up.  In  addition,  the  subject  does  not  have  to  give  consent  to  the 
surveillance.  Individuals  are  constantly  observed  by  cameras  with  or  without  their 
consent.  This  occurs  on  a  daily  basis  through  traffic,  automated  teller  machines,  and 
security  cameras. 

Problem  Statement 

To  utilize  recorded  video  of  individuals  walking  at  a  90-degree  angle  to  a  fixed 
camera  position  and  remove  the  shadow  the  individual  projects  on  surfaces,  such  as  the 
ground,  without  losing  a  significant  amount  of  data  related  to  the  individual. 

Research  Objectives 

The  objective  of  this  thesis  is  to  find  the  relationship  between  colors  and  their 

shaded  counterparts.  Once  this  relationship  is  derived  mathematically,  the  algorithm  will 
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remove  shaded  colors.  The  algorithm  needs  to  be  concise  to  eliminate  errors  in 
processing  and  broad  enough  to  remove  as  many  shadow  pixels  as  possible  without 
compromising  the  integrity  of  the  data.  The  algorithm  will  then  be  employed  on 
prerecorded  video  of  an  individual  walking. 

The  shadow  pixel  removal  will  ensure  noise  reduction  from  the  video  before  a 
background  subtraction  method  is  utilized.  After  the  shadow  and  background  subtraction 
has  been  accomplished,  the  resulting  binary  image  needs  smoothing  to  reduce  errors  in 
processing.  The  smoothing  will  involve  algorithms  pre-built  in  MATLAB. 

The  algorithms  will  take  prerecorded  video  of  an  individual  walking  and  extract  a 
mathematical  representation  of  the  individual  depicted  in  the  video.  In  addition,  the 
constructs  derived  are  written  so  they  do  not  require  human  interaction.  Every  program, 
once  initiated,  runs  without  the  input  of  parameters,  except  the  images  themselves. 

Research  Question 

Is  it  possible  to  derive  an  algorithm  to  remove  shadow  from  a  video  feed  without 
compromising  the  integrity  of  objects  to  be  observed? 

Thesis  Organization 

The  second  chapter  discusses  the  accomplishments  and  theories  of  individuals 
working  on  similar  research.  The  ideas  discussed  illustrate  the  level  of  exploration  that 
researchers  have  put  forth  in  this  area  of  gait  biometrics.  In  addition,  the  beginnings  of 
gait  biometrics  and  practical  uses  are  communicated.  The  discussion  will  center  on  the 
work  done  by  researchers  in  the  area  of  noise  removal,  such  as  shadow  or  background. 
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Their  algorithms  to  accomplish  gait  recognition  will  be  briefly  highlighted  to  illustrate 
their  common  need  to  remove  noise  in  the  data. 

The  third  chapter  illustrates  the  methodology  employed.  The  collection  of  data 
will  be  thoroughly  discussed.  Explanations  of  all  MATLAB  algorithms  employed  to 
reach  conclusions  in  Appendix  A  will  be  included.  In  addition,  contained  in  the  chapter 
are  limitations  of  the  hypothesis  test  and  the  methods  utilized  to  derive  conclusions.  The 
hypothesis  tests  to  determine  shadow  pixel  feasibility  will  be  incorporated  in  the  third 
chapter. 

The  fourth  chapter  contains  the  results  and  analysis  of  the  data  processed. 
Discussions  of  the  mathematical  formulas  utilized  to  derive  the  MATLAB  algorithms  are 
also  contained  in  the  chapter.  Pictorial  representations  of  the  data  are  plotted  in  three 
dimensions  to  aid  in  the  visualization  of  the  properties  of  the  backgrounds  and  the  shaded 
pixels. 

The  fifth  chapter  contains  conclusions  derived  from  the  analysis  of  data.  In 
addition,  recommendations  for  further  research  are  discussed.  Finally,  there  will  be  a 
summary  of  findings. 
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II.  Literature  Review 


Chapter  Overview 

This  chapter  is  dedicated  to  the  past  research  accomplished  in  the  field  of  gait 
decomposition.  The  primary  focus  deals  with  the  findings  of  scientists  related  to 
processes  for  useable  data  extracted  from  video  feeds  of  individuals.  Researchers  employ 
varying  methods  for  gait  recognition  depending  on  particular  self  imposed  extraction 
methods. 

“Human  movement  analysis  aims  at  gathering  quantitative  infonnation  about  the 
mechanics  of  the  musculo-skeletal  system  during  the  execution  of  a  motor  task” 
(Cappozzo,  2006:186).  In  addition,  Merriam- Webster’s  online  dictionary  describes 
biometrics  as  “the  measurement  and  analysis  of  unique  physical  or  behavioral 
characteristics  (as  fingerprint  or  voice  patterns)  especially  as  a  means  of  verifying 
personal  identity”  (“Biometrics,”  2007).  Biometrics,  in  regards  to  the  human  gait,  has 
been  used  in  the  past  for  medical  and  gait  recognition  research. 

“All  humans  follow  the  same  basic  walking  pattern,  but  their  gaits  are 
influence[d]  by  functions  of  their  entire  musculo-skeletal  structure”  (Post,  2006: 1).  The 
individuality  of  walking  patterns  are  derived  in  part  from  height,  weight,  build,  injuries, 
shoes,  and  stress.  Some  gait  attributes  vary  from  day  to  day.  Objects  like  briefcases  and 
backpacks  hinder  researchers’  abilities  to  distinguish  individual’s  biometrics. 

Beginnings 

The  human  gait  can  be  broken  down  into  components.  An  online  medical 
dictionary  states  the  human  gait  can  be  broken  into  three  phases.  One  phase  is  the 
“stance  phase”  where  both  feet  are  holding  the  weight  of  the  person.  The  “swing  phase” 
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refers  to  the  time  when  a  person  has  all  of  their  weight  on  one  foot  and  the  other  is 
suspended  in  the  off  of  the  ground.  The  third  phase  is  an  inverted  swing  phase,  where  the 
opposite  foot  supports  the  weight  (“Gait,”  2007). 

The  medical  community  has  long  utilized  the  analysis  of  gait  to  determine 
muscular  or  skeletal  problems  with  patients.  A  conjunction  of  several  different  systems 
can  be  used  to  obtain  data  about  a  specific  gait  trial.  “Cameras,  magnetic  fields, 
electromyography  (EMG)  data,  and  force  plates  are  all  methods  for  used  for  data 
collection”  (Post,  2006:6).  These  items  normally  record  a  person’s  gait  biometric  under 
laboratory  conditions.  These  processes  seek  to  eliminate  noise  derived  from  parameters 
not  associated  with  the  individual  included  in  the  data  feed. 

This  type  of  data  collection  involves  the  collaboration  of  the  patient.  Researchers 
have  the  ability  to  record  massive  amounts  of  data  about  any  one  facet  or  a  whole 
multitude  of  parameters  involved  with  an  individual’s  gait  biometrics  (Post,  2006:5). 

With  the  data  obtained,  doctors  and  technicians  can  recommend  a  particular  treatment  or 
pinpoint  a  problem.  These  types  of  problems  originate  from  sports  injuries,  strokes  or  a 
multitude  of  other  factors.  With  aid  of  computer-based  gait  data,  therapists  can  measure 
progress  mathematically,  thereby  reducing  the  human  error.  This  human  error  can  come 
from  the  constant  repetitive  treatments  required  by  patients  to  return  to  normal  gait  (Post, 
2006:8). 

The  medical  community  has  almost  limitless  access  to  a  volunteered  patient.  The 
patients  are  self-motivated  to  give  medical  professionals  full  access  to  help  remedy  their 
particular  medical  condition.  However,  researchers  in  the  gait  recognition  field  utilizing 
field  video  data  generally  have  seconds  to  record  data  under  non-ideal  conditions. 
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Relevant  Research 


Non-ideal  conditions  constitute  a  noise  that  analysts  in  the  field  must  deal  with. 
Isolated  feature  extraction  is  an  integral  part  of  gait  decomposition.  Data  extracted  with 
less  noise  would  be  more  useful  to  researchers.  Several  concerns  arise  from  data 
collected  in  the  field.  All  subjects  walking  in  captured  video  will  appear  at  varying 
lengths  from  the  camera.  The  process  should  scale  the  image  accordingly  (Boulgouris, 
2006:971).  As  an  individual  crosses  the  frame  of  reference  for  the  camera,  their 
placement  in  the  frame  shifts  frame  right  to  left  or  left  to  right.  A  gait  detection 
algorithm  should  account  for  frame  alignment  (Boulgouris,  2006:971).  If  a  background 
subtraction  is  utilized,  the  algorithm  should  correct  for  imperfect  background  subtraction. 
In  addition,  even  with  the  advent  of  faster  and  faster  computers,  the  process  should 
require  low  computational  power  (Boulgouris,  2006:971).  These  areas  of  concern 
surround  the  building  of  general  algorithms,  however,  they  do  not  limit  the  means  to 
identify  individuals  through  mathematical  interpretations  of  data  captured  from  video 
surveillance. 

The  methods  of  biometric  gait  identification  can  be  divided  into  2  categories, 
model-based  and  motion-based  (Post,  2006:2).  When  researchers  take  a  predetermined 
composite  and  then  match  new  data  to  it  that  is  a  model-based  approach  (Post,  2006:2). 
Extrapolating  information  from  a  subject  and  analyzing  it  equates  itself  to  a  motion-based 
approach  (Post,  2006:3).  Both  approaches  seek  the  ability  to  categorize  and  compare  sets 
of  unrelated  data. 

Model-based  approaches  neglect  to  account  for  irregularities  in  gait  (Foster, 
2003:2490).  These  irregularities  include  the  subject  carrying  a  briefcase  or  a  backpack. 
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One  way  a  model-based  approach  could  account  for  irregularities  is  if  a  database  was 
created  for  each  type  (Foster,  2003:2490).  Of  course,  there  would  be  a  multitude  of 
irregularities  to  consider.  Moreover,  a  database  for  each  irregularity  would  need  to  be  as 
extensive  as  the  normal  gait  database.  This  type  of  approach  is  usually  mathematically 
intricate  (Foster,  2003:2490). 

Motion-based  approaches  try  to  remove  the  human  element  from  the  equation. 
Using  this  approach,  researchers  utilize  a  binary  silhouette  or  some  other  shape 
representation  to  extract  data.  The  extraction  can  be  done  through  the  motion  created  by 
the  object  or  direct  matching  with  a  database  of  silhouettes  (Post,  2006:3). 

Both  approaches  deal  with  similar  problems.  Researchers  determine  which 
variables  are  noise  related  and  need  to  be  removed  before  classifications  are  done.  The 
issue  of  shadows  derived  from  the  individual  subject  is  one  source  of  noise,  so  shadow 
removal  is  the  major  goal  of  this  thesis.  Shadows  normally  obstruct  leg  movement, 
depending  on  sun  placement.  The  darkness  of  the  shadow  also  varies  with  light  intensity. 
The  shadows  from  moving  clouds  will  also  add  noise  to  the  data.  Several  papers  have 
approaches  to  deal  with  the  problems  associated  with  this  noise  and  then  compare 
individual  biometrics  to  those  in  a  database. 

The  researchers  that  derived  biometric  comparison  algorithms  are  of  particular 
importance  to  gait  recognition  analysts.  This  research  works  towards  the  decomposition 
of  biometrics  for  use  in  a  later  recognition  program.  Nonetheless,  a  brief  description  of 
methods  used  to  facilitate  these  comparisons  will  be  discussed. 
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Processing  Without  Shadow  Removal 

Before  gait  comparison  algorithms  can  be  employed,  the  separation  of  individuals 
from  their  background  is  an  initial  goal  that  gait  recognition  algorithms  generally  seek  to 
obtain.  Through  a  technique  dubbed  “non-linear  point  distribution  model,”  researchers 
extracted  the  motion  of  the  human  form  from  video  of  a  moving  subject  (Bowden, 
2000:729).  Researchers  were  able  to  create  a  body  contour  using  an  individual’s 
outstretched  hands  and  head  (Bowden,  2000:73 1).  The  process  was  accomplished  via  a 
blue  screen  and  chroma  keying  (Bowden,  2000:732).  This  technique  is  similar  to  how 
the  backdrop  of  a  television  weathennan  is  made  to  appear  to  be  constantly  changing. 

This  process  worked  well  in  laboratory  conditions,  shadow  was  not  a  factor. 

Researchers  from  The  Hong  Kong  Polytechnic  University  avoided  the  removal  of 
shadow  through  the  incorporation  of  “motion  silhouette  contour  templates  (MSCTs)  and 
static  silhouette  templates  (SSTs)”  (Lam,  2006:2563).  The  silhouettes  obtained  for  this 
gait  comparison  algorithm  included  shadows.  The  images  were  scaled  to  128  by  88 
pixels  to  account  for  the  different  sizes  of  the  silhouettes  (Lam,  2006:2566).  The  MSCTs 
generated  from  the  data  are  representations  of  one  individuals  gait  cycle  (Lam, 
2006:2567).  The  SSTs  are  a  means  to  exclude  errors  in  comparisons  of  gait  (Lam, 
2006:2569).  A  background  subtraction  method  was  accomplished  to  render  silhouettes  in 
this  process.  The  gait  recognition  process  yielded  successful  matches  80  percent  of  the 
time  (Lam,  2006:2563). 

An  additional  technique  for  automatic  gait  analysis  involves  “masking  functions 
to  measure  area  as  a  time  varying  signal  from  a  sequence  of  silhouettes  of  a  walking 
subject”  (Foster,  2003:2489).  In  order  to  get  silhouettes  into  a  state  that  is  feasible  for 
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this  type  of  analysis,  the  silhouettes  need  to  be  processed.  The  process  employed  begins 
with  a  background  removal  algorithm.  Since  data  was  collected  in  laboratory  conditions, 
a  solid  backdrop  was  used  and  the  silhouette  was  extracted  through  a  chroma-key 
technique  (Foster,  2003:2491).  The  researchers  then  cropped  the  pieces  of  the  image 
above  the  walking  subject  and  the  solid  backdrop.  The  final  images  were  scaled  to  a  64 
by  64  pixel  box  to  account  for  distance  from  the  camera  (Foster,  2003:2491).  The  final 
data  obtained  for  comparison  was  derived  from  horizontal  and  vertical  masks  isolating 
particular  body  segments  (Foster,  2003:2492).  With  the  use  of  laboratory  conditions 
researchers  in  this  experiment  avoided  the  problem  of  shadow  removal. 

Processing  With  Shadow  Removal 

Shadow  pixels  are  classified  as  the  shadow  cast  by  the  object  on  surfaces 
or  the  shadow  projected  onto  the  object  itself  (Wang,  2004:649).  Researchers  work  with 
shaded  pixels  in  a  variety  of  ways.  Another  gait  decomposition  algorithm  utilizes 
manually  constructed  silhouettes  where  components  are  weighted  to  accomplish  gait  style 
matching  (Boulgouris,  2007:1766).  The  components  are  comprised  of  the  arms,  thighs, 
legs,  lower  torso,  and  head.  This  method  resulted  in  a  relatively  high  match  percentage. 
Researchers  were  able  to  match  a  prior  subject’s  gait  cycle  to  a  later  recorded  gait  cycle. 
This  algorithm  was  accomplished  after  silhouettes  were  manually  derived  from  data  and 
were  scaled  to  account  for  the  varying  heights  of  individuals  (Boulgouris,  2007:1764). 
Although  the  researchers  removed  the  shadow,  the  process  was  done  manually  which 
requires  to  many  man-hours  to  be  practical.  The  binary  silhouettes  were  derived  from  the 
University  of  South  Florida  Human  Identification  Gait  Challenge  data. 
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University  of  South  Florida  Gait  Challenge  data  is  a  database  of  “300  GB 
(gigabytes)  of  data  related  to  452  sequences  from  74  subjects”  (Phillips,  2006:137).  The 
purpose  of  the  experiment  was  to  create  a  massive  database  to  allow  researchers  to 
incorporate  their  respective  algorithms  utilizing  a  common  data  set.  Some  of  the  74 
individuals  were  recorded  inside,  outside,  and  on  different  days  using  different 
perspectives  and  wearing  different  clothes  (Phillips,  2006: 137).  The  images  were 
processed  using  a  background  subtraction  method  similar  to  those  previously  discussed  in 
research.  The  binary  images  were  cleaned  up  using  a  9  by  9  pyramidal  averaging  filter 
(Phillips,  2006: 137).  Boxes  were  manually  assigned  to  the  silhouettes  to  remove 
unnecessary  data.  The  researcher’s  algorithm  worked  well  when  the  factors  of  surface 
and  shoe  type  were  mitigated.  Shadow  removal  was  accomplished  through  a 
combination  of  manual  procedures  for  the  bounding  of  a  subject  and  automatic  filtering. 

One  approach  to  automatically  remove  a  subject’s  shadow  employs  an  “eigen- 
stance”  gait  model  (Liu,  2005: 170).  Each  ordered  frame  of  a  recorded  gait  is  projected 
into  a  subspace  created  by  a  Hidden  Markov  Model  (Liu,  2005:173).  A  new  silhouette  is 
created  with  the  original  silhouette  data  “projected  onto  the  matched  eigen-stance  model” 
(Liu,  2005: 173).  The  method  of  background  subtraction  made  use  of  “Gaussian 
statistical  models”  (Liu,  2005:173).  Researchers  calculated  the  red,  blue,  and  green 
(RGB)  data  of  each  pixel  to  include  mean  and  variance.  Utilizing  the  mean  and  variance 
information  the  researcher’s  algorithm  determined  whether  a  pixel  was  background  or 
not.  This  process  works  quite  well  with  the  absence  of  dark  shadows.  The  projections 
and  preloaded  body  shapes  allowed  researchers  to  remove  shadow  leaving  a  silhouette 
comprised  entirely  of  body  components. 
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Summary 

The  researchers  presented  in  this  chapter  incorporated  several  varied  ways  to  a 
subject  from  the  surrounding  background.  The  motive  for  their  research  centered  on  the 
final  objective  of  gait  recognition.  Many  utilized  background  subtraction  algorithms  and 
processed  the  silhouettes  according  to  the  requirements  of  their  algorithms.  Some 
researchers  set  their  test  subjects  in  blue  screen  rooms  or  ideal  laboratory  conditions  to 
mitigate  the  outside  influence  of  variables  like  shadow.  Methods  for  dealing  with 
shadow  removal  ranged  from  manual  deletion  to  incorporation  into  the  researchers’ 
respective  algorithms.  This  thesis  attempts  to  automate  the  process  of  shadow  removal  so 
that  the  entire  process  of  gait  recognition  may  be  likewise  automated  from  the  capture  of 
video  to  individual  discrimination. 


11 


ITT.  Methodology 


Chapter  Overview 

In  this  chapter,  descriptions  of  practices  taken  to  collect  data  for  this  thesis  are 
discussed.  The  problem  of  isolating  the  human  gait  comes  with  the  problem  of  isolating 
human  gait  from  the  background.  When  background  subtraction  is  performed,  the  human 
gait  as  well  as  the  shadow  of  the  individual  typically  remains  in  each  frame.  Therefore,  a 
methodology  for  removing  shadows  is  presented.  Furthermore,  all  MATLAB  algorithms 
utilized  to  derive  results  are  discussed.  Finally,  limitations  associated  with  the  algorithms 
and  processing  are  conveyed. 

Data  Collection 

The  test  data  collected  for  processing  originated  with  the  filming  of  three  test 
backgrounds.  Filming  was  accomplished  at  10:30  a.m.  to  diminish  morning  haze  and 
allow  for  longer  shadows.  Cloud  cover  was  minimal.  The  camera  in  this  experiment  was 
locked  onto  a  tripod  located  54.5  inches  above  the  ground  to  minimize  camera  shake. 

Three  different  solid  colored  18  by  12  inch  construction  paper  sheets  were  placed 
horizontally  on  a  concrete  surface  and  filmed  in  open  sunlight  and  under  shadow.  An 
opaque  plastic  can  was  placed  between  the  construction  paper  and  the  sun  to  produce 
shadow.  Several  seconds  of  video  were  captured  of  each  color  background  under  both 
sun  light  and  shadow  conditions 

Research  Step 

Researchers  have  dealt  with  the  removal  of  shadows  in  silhouettes  in  several 
ways.  The  manual  removal  of  shadow  runs  the  risk  of  removing  pertinent  data  and  is  left 
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to  an  individual’s  interpretation  of  the  data.  The  time  needed  to  accomplish  a  manual 
rendering  of  shadow  removal  makes  it  impractical  to  employ.  Alternatively,  some 
researchers  choose  to  incorporate  noise  left  after  imperfect  background  subtraction 
methods  into  their  algorithms.  This  thesis  will  develop  and  demonstrate  an  effective 
automated  removal  of  shadow  and  will  further  show  the  removal  of  shadow  may  be 
accomplished  before  a  background  subtraction  algorithm  is  used.  The  process  is 
described  in  Figure  1. 


Digital  Video  Acquisition 


True  Binary  Silhouette 
Figure  1.  True  Binary  Silhouette  Algorithm 


The  first  step  taken  in  the  desire  to  remove  shadows  was  to  detennine  the  effect  of 
shadow  on  each  of  the  three  dimensions  of  color.  Images  are  represented  by  varying 
degrees  of  red,  green,  and  blue  (RGB).  Red,  green,  and  blue  backgrounds  were  chosen 
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because  these  are  the  colors  whose  intensities,  relative  and  absolute,  are  represented  by 
positive  integers  up  to  255.  By  testing  the  effects  of  shadow  on  specific  pixels  located  in 
the  solid  backgrounds,  the  effect  of  shadow  can  be  derived  for  different  pixel  value 
combinations. 

Solid  colors  were  utilized  as  a  background  in  order  to  remove  as  many  variables 
as  possible  from  the  experiment.  A  researcher  taking  an  average  background  would  have 
to  account  for  the  movement  of  the  camera  and  the  blending  of  colors  with  a  multicolored 
background.  Pixels  with  wide  variations  in  color  may  reside  next  to  each  other  giving 
skewed  results.  The  separate  analyses  of  these  three  solid  backgrounds  showed  a 
correlation  utilized  to  predict  the  effect  of  shadow  in  a  multitude  of  situations. 

Algorithms  Developed 

In  order  to  extract  the  data  necessary  to  perform  the  analyses,  several  MATLAB 
algorithms  were  employed.  The  frames  were  extracted  from  video  using  the  MATLAB 
function  LoadVideo  that  is  located  in  Appendix  A.  The  image  array  is  then  saved  to  the 
computer  with  an  algorithm  similar  to  Writemulti  located  in  Appendix  A.  At  least  30 
frames  of  480  by  720  pixels  of  the  sunlight-covered  solid  colored  construction  paper 
were  recorded.  Over  100  frames  of  solid  background  completely  covered  in  shadow 
were  recorded.  Frames  with  the  shadow  partially  covering  the  background  were 
removed.  The  images  were  then  cropped  to  100  by  200  pixels  with  the  function  Crop  in 
Appendix  A.  This  was  done  to  remove  everything  except  the  solid  colored  construction 
paper  from  the  test  region.  The  function  Crop  also  writes  the  cropped  image  to  file. 

Once  the  images  were  cropped  and  saved,  the  data  was  processed.  To  facilitate 
the  visualization  of  the  data  represented  in  this  thesis,  several  algorithms  were 
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constructed.  The  function  DataPlots  constructs  a  three-dimensional  plot  of  the  pixel 
values  of  the  average  background  and  the  average  pixel  values  of  shaded  frames.  The 
function  RotationPlot  illustrates  the  effect  of  rotation  matrices  on  a  line  and  the 
ellipsoidal  rejection  region  of  the  hypothesis  test.  RGBBackground  is  an  algorithm 
designed  to  illustrate  the  level  of  intensities  of  the  three  components  of  a  picture  (Samler, 
2006:60).  These  algorithms  display  the  properties  of  data  acquired  and  computational 
algorithms  are  discussed  next. 

Computational  algorithms  were  derived  to  statistically  analyze  the  data.  The 
average  distances  of  shaded  pixels  from  a  predetermined  line  are  computed  via  the 
algorithm  DistanceValues.  The  algorithm  DistanceProjectedValues  outputs  the  distances 
of  a  projected  line  to  a  point  on  a  predetermined  line.  The  95  percent  upper  confidence 
intervals  of  the  distances  are  computed  with  the  function  Confid.  The  algorithm  Coeff 
outputs  the  coefficients  utilized  in  the  shadow  removal  algorithm.  The  bases  of  these 
algorithms  will  be  discussed  in  Chapter  IV. 

Once  the  outputs  were  derived  from  the  computational  algorithms,  these  outputs 
were  incorporated  into  an  algorithm  for  shadow  removal.  In  order  to  accomplish  the 
shadow  removal,  an  average  background  of  sunlight-covered  background  is  detennined 
via  MATLAB.  This  average  background  is  incorporated  into  several  algorithms.  Once 
the  average  background  is  derived,  the  algorithm  will  then  derive  an  ellipsoidal  test 
region.  If  the  values  of  a  pixel  are  found  to  lie  in  the  interior  of  the  ellipsoid,  the  specific 
pixel  will  be  replaced  with  the  value  of  the  corresponding  pixel  in  the  average  picture. 
The  MATLAB  code  ShadowTest  is  employed  on  the  test  data  to  remove  shaded  pixels. 
This  code  will  be  discussed  in  detail  in  the  Chapter  IV.  Since  the  shadow  removal 


15 


algorithm  was  built  to  accommodate  any  background,  it  is  capable  of  performing  the 
same  process  on  other  video  data. 

Hypothesis  Test 

A  hypothesis  test  is  utilized  to  accomplish  shadow  removal.  The  effect  of  the 
presence  of  a  shadow  on  a  particular  pixel  is  that  the  representative  values  in  the  RGB 
primary  colors  are  lowered.  This  is  due  to  the  fact  that  lower  relative  numbers  represent 
darker  colors.  The  lowering  of  the  values  of  each  of  the  RGB  components  is  dependent 
on  the  relationship  between  each  of  the  RGB  components.  For  example,  if  the  RGB 
components  are  relatively  equal,  shaded  pixels  follow  a  relatively  or  approaching  linear 
path  towards  the  origin.  If  the  RGB  components  are  not  relatively  equal,  the  shaded 
pixels  are  found  to  be  contained  in  an  ellipsoid  whose  center  is  a  linear  path  to  the  origin 
from  the  original  pixel  value.  The  relationship  between  the  RGB  values  of  shaded  pixels 
dictates  the  size  and  shape  of  an  ellipsoidal  rejection  region.  If  a  pixel  value  resides  in 
the  ellipsoidal  region,  it  is  considered  shadow  and  is  replaced  with  the  corresponding 
pixel  values  from  the  average  background. 

The  dimensions  of  the  ellipsoidal  rejection  region  will  be  derived  from  the  pixel 
values  of  each  frame  to  be  tested.  The  major  axis  of  the  ellipsoid  comes  from  the 
distance  of  a  pixel  value  in  RGB  coordinates  to  the  origin.  The  two  minor  axes  are  also 
derived  from  the  RGB  values  of  the  pixels. 

The  height  of  the  ellipsoid  is  derived  from  the  combination  of  blue/red  or 
blue/green  components.  The  width  of  the  test  area  corresponds  to  the  green/red 
component  pixel  values.  The  derivation  of  the  formula  to  achieve  the  correct  height  and 
width  of  the  ellipsoid  is  discussed  in  detail  in  Chapter  IV. 
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The  ellipsoidal  area  represents  the  rejection  region.  The  null  hypothesis  of  the 
hypothesis  test  is  the  statement,  “The  pixel  value  is  a  shadow  pixel.”  The  alternative 
hypothesis  is,  “The  pixel  value  is  not  a  shadow  pixel.”  If  a  particular  pixel  value  of  the 
frame  were  found  to  be  a  shadow  pixel,  it  is  replaced  with  the  corresponding  pixel  from 
the  average  background.  If  the  pixel  is  not  a  shadow  pixel,  it  is  unchanged. 

Limitations 

Inherent  in  all  hypothesis  tests  is  the  possible  misclassification  of  tested  data. 
Three  such  misclassifications  are  considered.  First,  the  hypothesis  test  can  deem 
background  pixels  as  shaded  pixels.  If  this  error  occurs,  the  background  pixel  is  replaced 
with  the  average  background  pixel.  This  type  of  error  does  not  result  in  any  loss  of 
pertinent  data. 

Second,  the  hypothesis  test  can  identify  part  of  the  individual  as  a  shaded  pixel.  If 
this  type  of  error  occurs  the  result  would  be  a  loss  of  data  needed  to  create  a  binary 
silhouette.  This  type  of  error  would  occur  if  the  pixel  value  of  the  person’s  hair,  skin,  or 
clothing  resided  in  the  ellipsoid  test  region.  Small  holes  in  the  silhouette  can  be  remedied 
with  built-in  MATLAB  functions.  In  contrast,  large  cuts  in  the  silhouette  have  little 
chance  of  remediation  without  manual  intervention. 

Third,  it  is  possible  for  the  value  of  a  shadow  pixel  to  be  deemed  a  non-shadow 
pixel.  This  type  of  error  would  occur  if  the  natural  variation  in  a  pixel’s  RGB  values 
placed  it  outside  the  ellipsoidal  rejection  region.  If  the  number  of  unreplaced  shadow 
pixels  is  sufficiently  low,  built-in  smoothing  functions  in  MATLAB  may  remove  the 
small  percentage  of  errors. 
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Other  types  of  errors  are  inherent  to  the  algorithms  employed.  If  a  background 
contains  moving  objects,  these  objects  may  be  identified  as  silhouettes.  For  example,  a 
video  of  a  crowded  street  may  not  be  an  ideal  setting.  In  addition,  a  person  walking  in 
the  opposite  direction  of  the  intended  target  poses  a  difficult  problem  to  a  background 
subtraction  method. 

Furthermore,  the  ellipsoidal  rejection  region  utilized  for  the  hypothesis  test  is 
derived  from  two  rotation  matrices.  If  the  coordinates  of  a  pixel  lie  along  two  of  the  three 
axes,  gimbal  lock  may  occur.  This  refers  to  the  problem  of  when  the  two  rotation 
matrices  approach  90  degrees;  the  third  axis  cannot  be  accounted  for  or  changed.  This 
error  is  deemed  unlikely,  however,  since  on  nearly  all  real  world  situations,  color  is 
composed,  to  some  degree,  of  all  three  considered  colors,  red,  green,  and  blue. 

The  limitations  of  the  hypothesis  test  are  complicated  by  the  limitations  of  the 
camera  utilized  for  data  collection.  The  number  of  pixels  recorded  by  the  camera  for 
each  frame  is  directly  correlated  to  the  quality  of  the  resulting  silhouettes.  In  addition, 
the  ability  of  the  camera  to  accurately  interpret  its  usual  field  of  view  is  imperative  to 
silhouette  resolution. 
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IV.  Results  and  Analysis 


Chapter  Overview 

This  chapter  will  examine  the  analysis  and  results  obtained  through  research  of 
frames  of  three  solid  colored  backgrounds,  as  well  as  frames  of  actual  gait  data.  In  order 
to  deconstruct  an  individual’s  gait,  elements  of  the  video  other  than  the  individual  studied 
need  to  be  removed.  This  includes  the  visible  background  and  shadow  produced  from  the 
blockage  of  light.  Previous  work  has  successfully  removed  the  background  with  some 
residual  noise  in  the  data.  Here  is  presented  a  viable  methodology  to  remove  the  shadow 
using  the  test  data  of  three  solid  colored  backgrounds.  The  method  is  then  implemented 
on  video  data  containing  an  individual  walking  perpendicular  to  the  camera  (Samler, 
2006:27).  This  data  will  be  referred  to  as  gait  data.  The  test  data  of  the  three  solid 
colored  backgrounds  was  collected  by  the  current  researcher  while  the  gait  data  of 
individuals  walking  across  the  frame  was  collected  by  Samler  (Samler,  2006:27). 

Analysis  of  Test  Data 

Test  data  was  recorded  on  a  day  when  conditions  were  optimal  for  an 
unobstructed  view  of  the  sun.  Recordings  were  accomplished  when  the  shadow  was  of  a 
size  adequate  to  completely  cover  the  solid  background  material.  In  addition,  in  order  to 
minimize  the  effect  of  uncontrollable  shadow  fluctuations,  recording  was  accomplished 
when  clouds  were  sparse. 

Minimized  environmental  conditions  are  one  aspect  of  the  controlled  experiment. 
Solid  backgrounds  will  constitute  another  aspect  of  the  controlled  experiment.  The  three 
colors  of  the  solid  backgrounds  were  red,  green,  and  blue.  They  were  chosen  because 
joint  photographic  experts  group  (  jpeg)  files  are  stored  in  a  RGB  format.  Individual 
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frames  were  stored  to  computer  in  a  jpeg  format  utilizing  MATLAB.  Each  frame  of  data 
was  stored  in  a  480  by  720  by  3  format.  This  relates  to  three  480  by  720  pixel  frames 
representing  the  RGB  colors.  The  450  test  data  frames  collected  were  cropped  to  100  by 
200  in  order  to  contain  the  maximum  amount  of  solid  background.  Each  of  the  red,  blue, 
and  green  backgrounds  is  analyzed  separately,  and  the  results  are  used  to  drive  a 
hypothesis  test  to  effect  the  removal  in  the  gait  data. 

Blue  Background  Analysis 

The  results  from  the  150  blue  background  frames  are  analyzed  first.  Due  to  minor 
variability  in  the  pixel  values,  it  is  more  feasible  to  work  with  the  average  of  several 
frames  without  shadow  visible.  This  variability  may  be  the  result  of  ripples  in  the 
construction  paper  or  camera  shift.  Therefore  comparisons  of  shadow  covered  individual 
frames  will  be  done  with  the  average  sunlit  frame.  After  these  images  were  recorded,  a 
solid  object  was  placed  between  the  sunlight  and  the  construction  material.  Further  video 
was  recorded  with  a  shadow  covering  the  blue  background.  The  average  blue 
background  of  the  first  twenty  frames  and  a  typical  shaded  frame  is  shown  in  Figure  2. 
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Figure  2.  Blue  Average  Background  and  Typical  Shaded  Frame 
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Figure  3  displays  the  average  individual  RGB  intensities  of  the  blue  average 
background  and  a  typical  shaded  frame.  The  smallest  average  pixel  value  represents  the 
least  intense  color.  Since  the  figure  depicts  a  strictly  blue  background,  the  tallest  bar  in 
each  chart  corresponds  to  the  blue  component.  The  background  color  is  visually  affected 
by  the  introduction  of  shadow.  In  addition,  the  intensities  of  the  average  RGB 
components  are  lessened  by  the  introduction  of  shading. 


Bar  Graph  of  Average  Values  of  RGB  Components  for  Blue  Average  Background 
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Figure  3.  Intensities  of  Average  Blue  Frame  and  Typical  Shaded  Blue  Frame 


Table  1  displays  pixel  values  in  the  RGB  spectrum  for  10  randomly  selected 
pixels.  It  is  shown  that  the  intensities  of  each  of  these  RGB  component  pixel  values  are 
significantly  reduced.  This  is  due  to  the  level  of  the  shadow  and  the  placement  of  the 
original  average  pixel  without  shadow.  Intuitively,  the  darker  the  shadow,  the  closer  a 
pixel’s  value  approaches  (0,0,0).  The  mean  and  standard  deviation  are  derived  from  the 
entire  population. 
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Table  1.  Sample  Pixel  Values  for  Blue  Background  With  Population  Statistics 


Red  Com 

ponent 

Green  Component 

Blue  Component 

No  Shadow 

Shadow 

No  Shadow 

Shadow 

No  Shadow 

Shadow 

112 

70 

130 

79 

224 

154 

116 

65 

135 

82 

230 

160 

125 

66 

144 

78 

242 

154 

116 

62 

136 

73 

231 

139 

120 

65 

139 

78 

235 

148 

122 

65 

141 

74 

236 

143 

120 

67 

140 

80 

235 

159 

118 

69 

136 

79 

230 

164 

121 

70 

139 

82 

234 

164 

121 

72 

141 

85 

238 

164 

Mean 

120.63 

67.54 

139.53 

80.60 

234.91 

156.65 

St.  Dev. 

1.63 

0.33 

1.74 

0.36 

2.32 

0.63 

To  help  with  the  interpretation  of  the  data  from  Table  2,  a  three-dimensional  plot 
of  one  hundred  randomly  selected  pixels  from  shaded  frames  are  displayed  in  Figure  4. 
Only  twenty  randomly  selected  pixels  were  plotted  from  the  average  frame. 


Example  of  blue  background 


Figure  4.  Randomly  Selected  Pixel  Values  for  the  Blue  Frames 
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Green  Background  Analysis 

The  green  background  is  analyzed  next.  Figure  5  shows  the  average  green 
background  for  the  first  42  frames  and  a  typical  shaded  green  background.  Frames  43 
and  higher  contain  shaded  pixels  and  are  infeasible  to  include  in  the  average  background. 
The  shaded  frame  is  noticeable  darker  than  the  average  green  background. 


Figure  5.  Green  Average  Background  and  Typical  Shaded  Frame 


Figure  6  illustrates  the  average  intensities  of  the  RGB  components  of  the  average 
green  frame  and  a  typical  shaded  green  frame.  The  green  dimension  is  the  most  intense 
since  the  background  is  green.  In  contrast  to  the  blue  test  data,  the  green  material  used 
was  not  as  purely  green  as  the  blue  background.  The  average  intensities  of  the  shaded 
frame  are  darkened  due  to  the  effect  of  shadow.  The  green  component  remains  the  most 
intense. 
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Bar  Graph  of  Average  Values  of  RGB  Components  for  Green  Average  Background  Bar  Graph  of  Average  Values  of  RGB  Components  for  Green  Shaded  Background 
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Figure  6.  Intensities  of  Average  Green  Frame  and  Typical  Shaded  Green  Frame 


Table  3  is  a  representation  of  ten  randomly  selected  pixels  taken  from  the  average 
green  frame  and  the  shaded  green  frame.  The  means  and  standard  deviations  were 
derived  from  the  entire  population  of  pixels. 


Table  2.  Randomly  Selected  Pixel  Values  for  Green  Background 


Red  Component 

Green  Component 

Blue  Component 

No  Shadow 

Shadow 

No  Shadow 

Shadow 

No  Shadow 

Shadow 

88 

53 

162 

91 

105 

76 

87 

56 

158 

97 

103 

81 

87 

52 

162 

96 

105 

81 

85 

54 

159 

97 

103 

80 

88 

53 

161 

94 

105 

78 

85 

53 

161 

94 

103 

78 

88 

53 

158 

91 

103 

76 

88 

54 

165 

98 

106 

81 

88 

51 

163 

95 

106 

78 

87 

54 

162 

98 

105 

81 

Mean 

88.30 

53.06 

161.58 

94.52 

105.39 

78.34 

St.  Dev. 

0.26 

0.96 

0.49 

1.48 

0.39 

1.26 

Figure  7  illustrates  data  taken  from  collected  green  frame  data.  One  hundred 
randomly  chosen  pixels  from  shaded  green  frames  were  plotted.  Twenty  pixel  values 
were  plotted  from  the  average  green  frame. 
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Example  of  Green  Background 


Figure  7.  Randomly  Selected  Pixel  Values  for  the  Green  Frames 

Red  Background  Analysis 

The  final  test  data  background  color  is  red.  The  background  is  not  one  solid  color 
due  to  small  ripples  in  the  construction  paper  background.  The  average  background  of 
the  first  37  frames  and  a  typical  red  shaded  background  are  shown  in  Figure  8.  Visually, 
the  colors  of  the  shaded  background  are  more  subdued. 
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Figure  8.  Red  Average  Background  and  Typical  Shaded  Frame 
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Figure  9  displays  the  average  RGB  intensities  of  the  average  red  frame  and 
typical  shaded  red  frame.  In  this  case,  the  red  component  has  the  highest  intensity 
followed  by  the  blue.  The  green  component  is  the  least  intense  due  to  the  color  of  the 
construction  paper.  The  green  and  red  components  are  affected  more  by  shadow  than  the 
blue  component. 


Bar  Graph  of  Average  Values  of  RGB  Components  for  Red  Average  Background 
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Bar  Graph  of  Average  Values  of  RGB  Components  for  Red  Shaded  Background 


Component 


Figure  9.  Intensities  of  Average  Red  Frame  and  Typical  Shaded  Red  Frame 


Table  3  is  a  random  representation  of  ten  pixels  taken  from  the  average  frame  and 
the  shaded  red  frames.  The  mean  and  standard  deviation  derived  from  the  entire 
population  of  pixels  is  also  located  in  the  table. 
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Table  3.  Randomly  Selected  Pixel  Values  for  the  Red  Frames 


Red  Component 

Green  Component 

Blue  Component 

No  Shadow 

Shadow 

No  Shadow 

Shadow 

No  Shadow 

Shadow 

228 

140 

93 

63 

134 

107 

227 

144 

93 

65 

133 

110 

228 

143 

92 

64 

133 

111 

223 

147 

90 

64 

130 

110 

220 

144 

86 

61 

126 

107 

229 

146 

93 

67 

134 

114 

225 

146 

91 

64 

131 

112 

226 

147 

91 

65 

131 

113 

226 

148 

92 

64 

132 

114 

226 

148 

91 

66 

132 

114 

Mean 

224.70 

150.36 

90.24 

66.83 

130.57 

114.95 

St.  Dev. 

0.76 

0.96 

0.47 

0.67 

0.70 

0.92 

In  order  to  better  visualize  this  data,  Figure  10  is  a  three-dimensional  graph  of  one 
hundred  randomly  chosen  points  from  a  shaded  red  background  frame  and  twenty  points 
from  the  average  sunlit  frame. 


Example  of  Red  Background 


Figure  10.  Randomly  Selected  Pixel  Values  for  the  Red  Frames 
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Analysis  of  All  Test  Data 

The  data  in  all  of  the  three-dimensional  plots  show  a  distinct  relationship  between 
the  shaded  pixel  values  and  the  average  background  pixel  values.  As  a  shadow  is  applied 
to  a  pixel,  the  corresponding  RGB  values  of  that  pixel  decrease.  The  observed  amount  of 
decrease  is  dependent  on  the  individual  RGB  component.  The  blue  component,  in  all 
three  test  cases,  decreased  less  substantially  than  the  red  and  green  components. 
Intuitively  the  darkest  a  pixel  value  of  a  shadow  can  be  is  the  value  associated  with  the 
color  black,  which  is  represented  with  [0,0,0]  in  the  RGB  dimensional  space.  With  the 
two  ends  of  the  major  axis  being  the  origin  and  the  pixel  value  of  the  average 
background,  and  with  an  intennediate  set  of  points  representing  the  shaded  pixels,  the 
equation  for  an  ellipsoid  encompassing  these  intennediate  set  of  shaded  pixels  can  be 
derived.  This  is  the  basis  of  the  rejection  region  utilized  in  the  hypothesis  test  for  this 
thesis. 

In  order  to  build  the  ellipsoidal  rejection  region,  the  parameters  for  an  ellipsoid 
need  to  be  solved  for.  Six  parameters  are  needed  to  define  a  unique  ellipsoid.  These 
parameters  are  the  coordinates  of  the  center  point  and  the  lengths  of  the  major  and  two 
minor  axes.  The  ellipsoid  equation  is: 


In  order  for  the  rejection  region  of  the  hypothesis  test  to  be  employed  without 

human  intervention,  the  parameters  for  the  ellipsoid  need  to  be  derived  from  the  pixel 

values  of  the  average  background.  The  u  variables  correspond  to  the  coordinates  of  the 
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center  point  of  the  ellipsoid.  Since  the  ends  of  the  major  axis  are  the  average  background 
pixel  value  and  the  origin,  the  center  point  of  the  ellipse  is  the  midpoint  of  the  line 
formed  by  these  points.  The  major  axis  of  the  ellipsoid,  a,  is  the  length  of  the  line 
segment  from  the  origin  to  the  value  of  the  pixel  in  the  average  photo  which  will  be 
referred  to  as  the  ellipsoid  centerline.  The  last  two  variables,  b  and  c,  are  derived  using 
the  values  of  the  shadow  pixels  in  the  test  data.  To  find  the  two  minor  axes  lengths, 
lengths  that  are  hereafter  termed  the  height  and  width  of  the  ellipsoid,  test  data  frames 
will  be  analyzed.  Examining  the  distance  of  the  values  of  the  shadow  pixels  from  a  line 
created  by  average  background  pixel  values  and  the  origin  reveals  the  distribution  of  the 
parameters  of  the  ellipsoid.  Figure  1 1  displays  each  RGB  histogram  of  all  of  the  three- 
dimensional  distances  of  the  shaded  pixels  to  the  ellipsoid  centerline. 
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x  io4  Histogram  of  Distances  from  Shadowed  Pixel  to  3-d  Line  Red  Frame 


Distance  of  Pixel  from  3-d  Line 


x  iQ5Histogram  of  Distances  from  Shadowed  Pixel  to  3-d  Line  Green  Frame 


Distance  of  Pixel  from  3-d  Line 


x  iQ5  Histogram  of  Distances  from  Shadowed  Pixel  to  3-d  Line  Blue  Frame 


Distance  of  Pixel  from  3-d  Line 


Figure  11.  RGB  Histograms  of  Distances  of  Shaded  Pixels  to  Centerline 


The  three-dimensional  distances  of  the  shaded  pixels  to  the  ellipsoid  centerline 
reveal  shadow  pixel  distances  but  not  where  the  pixels  are  located  in  relation  to  the 
centerline.  Analyzing  the  two  dimensional  combinations  of  each  RGB  test  frame  would 
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reveal  more  insight  into  the  orientation  of  the  shaded  pixel  values  in  relation  to  the 


centerline. 

The  parameters  of  the  ellipsoid  need  to  be  derived  from  the  two-dimensional 
projected  linear  distances  of  the  shaded  pixel  values  to  the  mid-point  of  the  line  created 
by  the  average  background  pixels  and  the  origin  (centerline).  There  are  three  two- 
dimensional  cases  to  discuss.  The  red/blue,  green/blue  and  red/green  components  of  the 
pixel  values  describe  all  combinations  of  three-dimensional  representations. 

The  means  and  standard  deviations  of  the  two-dimensional  distances  of  the 
shadow  pixel  values  in  relation  to  the  ellipsoid  centerline  are  in  Table  4.  The  data  was 
derived  running  the  DistanceValues  MATLAB  algorithm  in  Appendix  A.  Analyzing 
Table  4  reveals  that  the  blue  component  of  the  shadow  pixels  is  responsible  for  the 
largest  distances.  Therefore,  the  ellipsoid  height,  corresponding  to  the  blue  axis,  will  be 
derived  from  combinations  of  blue/green  or  red/blue.  The  width  of  the  ellipsoid  will 
come  from  the  combination  of  red/green  components. 


Table  4.  Two-Dimensional  Distances  of  Shadow  Pixels  to  Ellipsoid  Centerline 


Red  Frame 

Green  Frame 

Blue  Frame 

Mean 

Std 

Mean 

Std 

Mean 

Std 

Red/Blue 

22.1125 

2.0074 

9.6296 

1.5538 

12.2105 

1 .8484 

Blue/Green 

10.6215 

0.7989 

13.9437 

1.4901 

10.3804 

1.3814 

Red/Green 

4.4053 

1.7505 

1.6141 

1.3314 

2.6216 

1.6968 

The  distances  recorded  in  Table  4  were  recorded  at  varying  distances  along  the 
ellipsoid  centerline.  The  two  minor  axes  of  the  test  ellipsoid  can  be  thought  of  as 
emanating  from  the  center  of  the  ellipsoid.  In  order  to  get  a  shaded  pixel  distance  from 
the  center  point  of  the  ellipsoid,  a  three-dimensional  line  is  projected  from  the  average 
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background  pixel  value  through  the  shaded  pixel  value,  referred  to  as  a  shadow  line.  The 
tangent  of  the  angle  between  the  two  lines  is  calculated.  Taking  the  tangent  of  the  angle 
and  the  distance  from  the  average  background  pixel  values  to  the  center  point  of  the 
ellipsoid  reveals  the  orthogonal  distance  of  the  center  point  of  the  ellipsoid  to  the 
projected  shadow  line.  Figures  12  through  14  display  the  concept  discussed. 


Figure  12.  2D  Plots  of  Projected  Shadow  Lines  of  Blue  Background 
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Figure  13.  2D  Plots  of  Projected  Shadow  Lines  of  Green  Background 
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Figure  14.  2D  Plots  of  Projected  Shadow  Lines  of  Red  Background 


32 


The  distances  of  the  projected  shadow  line  to  the  center  of  the  ellipsoid  and  the 


charts  in  Figures  12  through  14  were  created  in  the  MATLAB  program 
DistanceProjectedValues  contained  in  Appendix  A.  To  encapsulate  the  maximum 
amount  of  shadow  pixels  an  upper  95%  confidence  interval  of  distances  from  the 
ellipsoid  center  point  to  the  projected  shadow  line  is  utilized.  Table  5  shows  the  average 
distances  of  the  first  ten  frames  of  shadow  pixels  from  the  center  point  of  the  ellipsoid. 
Table  6  illustrates  the  upper  95%  confidence  intervals  of  all  data  collected.  The 
information  presented  in  Table  6  was  output  from  the  MATLAB  program  Confid  in 
Appendix  A. 


Table  5.  Average  Distances  of  First  Ten  Shaded  Frames 


Red  Frame 

Green  Frame 

Blue  Frame 

Red/Gr 

Red/BI 

Blue/Gr 

Red/Gr 

Red/BI 

Blue/Gr 

Red/Gr 

Red/BI 

Blue/Gr 

8.28 

37.21 

13.47 

1.72 

20.60 

18.37 

2.35 

11.53 

9.92 

8.63 

38.14 

12.96 

1.72 

20.65 

18.41 

1.80 

11.27 

10.20 

8.13 

36.90 

13.55 

1.61 

20.88 

18.47 

1.91 

11.33 

10.16 

8.53 

37.91 

12.16 

1.68 

20.91 

18.56 

1.42 

11.27 

10.57 

8.40 

37.78 

13.13 

1.67 

20.45 

18.22 

1.84 

11.46 

10.36 

8.42 

38.60 

12.15 

1.47 

20.83 

18.31 

1.88 

11.41 

10.27 

8.99 

39.31 

12.35 

1.69 

20.50 

18.28 

1.81 

11.23 

10.15 

8.99 

39.93 

12.55 

1.70 

20.74 

18.46 

1.65 

11.03 

10.10 

10.21 

42.84 

13.11 

1.40 

20.85 

18.28 

2.07 

11.48 

10.15 

9.46 

42.40 

12.92 

1.64 

21.14 

18.54 

1.83 

11.29 

10.03 

Table  6.  Upper  Confidence  Intervals 


Red  Frame 

Green  Frame 

Blue  Frame 

Mean 

Mean 

Mean 

Red/Blue 

43.2667 

21.3211 

13.2447 

Red/Green 

11.3852 

2.2839 

3.1505 

Blue/Green 

15.5642 

18.7318 

11.5069 

Utilizing  the  distances  for  the  RGB  test  data,  an  ellipsoid  can  be  built  to  encompass 
the  shadow  pixels  tailored  to  each  particular  frame.  However,  once  the  values  are  applied 
in  different  background  color  settings,  the  results  are  mixed.  It  is  considered  that  the 
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ratios  of  the  red,  blue,  and  green  components  play  an  integral  part  in  the  values  of  shadow 
pixels. 

The  two-dimensional  component  combinations  in  each  frame  and  the  correlated 
upper  confidence  intervals  are  utilized  to  find  their  relationship.  Three  equations  are 
constructed  to  account  for  the  red/blue,  red/green,  and  blue/green  component 
combinations.  Coefficients  a  and  b  are  solved  for  the  three  combinations. 

a  (component  value)  +  b  (component  value)  =  Upper  Confidence  Interval  (2) 

The  a  and  b  coefficient  for  the  red/blue  components  are  [0.2467;  0.0584].  The 
coefficients  for  the  blue/green  components  are  [0.1294;  0.0242].  The  red/green 
coefficients  are  [0.0783;  0.0512].  When  an  average  background  pixel  value  is  introduced 
to  the  shadow  removal  algorithm,  the  respective  pixel  values  of  the  color  combinations 
are  multiplied  by  their  components  and  the  larger  of  the  two  values  is  set  to  the  height  of 
the  ellipsoid  test  region.  With  these  coefficients,  the  parameter  for  the  width  minor  axis 
can  be  detennined.  An  example  of  a  typical  ellipsoid  for  testing  can  be  seen  in  Figure  15. 
The  height  of  the  ellipsoid  was  set  to  ten  and  the  width  was  set  to  five  to  illustrate  that  the 
height  of  the  ellipsoid  is  greater  than  the  width. 
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Example  of  Test  Area  Ellipsoid 


Figure  15.  Example  of  Ellipsoid  Test  Region 

The  height  and  width  of  the  critical  region  have  been  derived.  In  contrast  the 
orientation  of  the  critical  region  is  vital  to  proper  implementation.  In  order  to  orient  the 
critical  region,  Euler  angles  and  rotation  matrices  were  employed.  The  ellipsoid  equation 
is  changed  to  matrix  fonn  to  aid  in  the  use  of  rotation  matrices.  Equation  3  illustrated  the 
ellipsoid  equation  in  matrix  form.  Equations  4  and  5  are  the  representations  of  the 
parameters  of  Equation  3.  The  x  and  u  in  Equation  4  are  vectors  of  the  three-dimensional 
components.  The  matrices  in  Equation  6  are  the  rotation  matrices  utilized  to  position  the 
test  region.  Equation  7  is  the  completed  rotation  matrix.  The  order  of  the  rotation 
matrices  is  important.  Equation  8  is  the  final  matrix  form  of  the  ellipsoid  critical  region 
with  rotation  matrices  incorporated. 


XTVX  =  1  (3) 

X  =  ((x)-(u))  (4) 
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(5) 


1/ a1  0  0 

V=  0  Mb1  0 
0  0  1  tc 


cos  6  0  sin  6 

0  1  0 
-sin#  0  cos# 
cos#  sin#  0 
-sin#  cos#  0 
0  0  1 

R  =  RyR: 

XtRtVRX  =  1 

The  rotation  matrices  are  derived  from  data  taken  from  the  endpoint  of  the 
ellipsoid.  A  rotation  about  the  blue  axis  is  accomplished  followed  by  a  rotation  on  the 
green  axis  in  order  to  orient  the  major  axis  of  the  ellipsoid  along  the  line  segment  defined 
by  the  origin  and  the  average  pixel  values.  Figure  16  illustrates  the  procedure.  The  line 
parallel  to  the  red  axis  is  the  start  line,  which  represents  the  original  line  before  rotation. 
The  angle  between  the  start  line  and  the  line  in  the  red/green  plane  is  calculated.  The  sine 
and  the  cosine  of  the  angle  is  inserted  Rv.  Next,  the  angle  between  the  line  in  the 
red/green  plane  and  the  center  of  the  ellipsoid  is  calculated.  In  order  to  make  this  next 
transformation,  only  the  sine  of  the  angle  is  utilized  and  the  cosine  of  the  angle  is  set  to  1. 
They  are  utilized  in  the  R:  transfonnation  matrix. 


(7) 

(8) 


36 


Transformations  of  Test  Area  Ellipsoid 


Figure  16.  Representation  of  Transformations  of  Ellipsoid  Test  Region 

Implementation 

Having  derived  the  equations  to  build  the  ellipsoidal  critical  region,  utilization  of 
the  test  on  the  solid  colored  test  data  was  perfonned.  Utilizing  the  ellipsoid  rejection 
region,  over  90  percent  of  the  shadow  pixels  were  removed  from  the  blue  background. 
Approximately  95  percent  of  the  shadow  pixel  values  in  the  green  background  were 
detected.  The  shadow  pixels  in  the  red  background  were  detected  99  percent  of  the  time. 
The  percentages  were  not  100  percent  due  to  the  values  of  shadow  pixels  in  three- 
dimensions.  For  example,  if  all  three  RGB  values  were  outside  of  the  95  percent 
confidence  interval  of  the  critical  region  they  would  not  get  detected  as  a  shadow  pixel. 
The  purpose  of  the  algorithm  is  to  eliminate  a  high  percentage  of  shaded  pixels. 
Although  the  results  were  reasonable,  the  program  does  allow  for  the  tightening  of 
parameters  to  ensure  that  objects  are  not  removed  that  need  to  be  studied.  These  solid 
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colored  backgrounds  were  merely  used  to  test  the  developed  methodology  in  preparation 
for  its  use  on  gait  data  (Samler,  2006:60). 

The  true  test  of  the  algorithm  comes  from  the  removal  of  shadow  without 
compromising  the  integrity  of  the  rest  of  the  image.  As  a  point  of  reference  for  the  rest  of 
the  discussion  of  results,  the  background  subtraction  method  employed  by  Samler 
resulted  in  silhouettes  in  Figure  17  (Samler,  2006:60). 


Figure  17.  Images  Derived  from  Background  Subtraction 

Frames  55,  56,  and  57  of  a  single  subject  from  the  gait  data  were  selected  due  to 
the  position  of  the  subject  crossing  the  field  of  view.  The  subject  is  in  full  view  of  the 
camera.  The  binary  silhouettes  in  Figure  18  represent  an  image  with  a  significant  amount 
of  noise.  There  are  several  MATLAB  filters  that  can  help  remove  much  of  this  noise. 
Unfortunately,  the  filters  are  ineffective  at  removing  the  shadow.  The  developed 
methodology  of  shadow  removal  via  the  ellipsoid  test  is  utilized  on  the  same  three  images 
and  then  the  background  subtraction  algorithm  is  run.  The  results  are  in  Figure  18. 
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Figure  18.  Images  After  Ellipsoid  Test  and  Background  Subtraction 


The  noise  in  the  three  frames  is  slightly  reduced.  Of  significance  is  the  fact  that 
the  area  around  the  subjects  feet  covered  in  shadow  now  appears  shadow-free,  albeit  still 
noisy.  In  addition,  there  are  portions  of  the  silhouette  that  have  been  removed.  Data  of 
an  individual  is  lost  when  a  pixel  value  is  in  the  critical  region.  This  happens  when  an 
individual’s  clothing  resembles  that  of  the  average  background  pixel  value.  Once  a 
segment  of  the  silhouette  has  been  removed,  that  data  of  an  individual  is  lost.  If  the  data 
lost  are  interior  points,  a  fill  command  may  rectify  the  problem.  However,  if  the  pixels 
lost  are  boundary  pixels  on  the  silhouette,  the  size  and  shape  will  be  altered.  Depending 
on  the  amount  of  data  lost,  this  process  can  significantly  or  minutely  effect  future 
calculations  derived  from  output  silhouettes. 

MATLAB  contains  built-in  functions  to  help  enhance  image  quality.  Utilizing  the 
MATLAB  built-in  filter,  medfilt2,  the  noise  can  be  removed  while  maintaining  the 
integrity  of  the  silhouette.  The  medfilt2  filter  takes  the  integer  value  of  a  determined 
neighborhood  and  returns  the  integer  median  value.  Figure  19  was  derived  utilizing  a  20 
by  20  neighborhood. 
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Figure  19.  Images  After  Ellipsoid  Test  and  Background  Subtraction  with  Filter 

The  image  is  still  in  need  of  certain  corrections.  The  hole  in  the  torso  area  was  an 
error.  This  was  due  to  the  color  of  the  subject’s  shorts.  The  color  of  the  subject’s  shorts 
closely  resembled  the  average  background  and  the  shadow  produced  by  the  outstretched 
ann  on  the  shorts  created  pixel  values  residing  in  the  ellipsoidal  region.  Since  the 
missing  data  is  a  hole  in  the  silhouette,  another  MATLAB  function  can  be  employed  to 
handle  errors  of  this  type.  The  command  imfill  fills  in  holes  in  binary  objects.  Figure  20 
is  the  representation  of  the  same  three  silhouettes  after  the  imfill  command  has  been 
enacted. 
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Figure  20.  Silhouette  with  Holes  Filled 
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Summary 

The  effect  of  shadow  on  different  combinations  of  colors  can  be  mathematically 
represented.  With  this  mathematical  representation,  the  shadow  pixels  that  belong  to  a 
corresponding  color  may  be  isolated  and  removed.  The  RBG  test  frames  analyzed  in  this 
chapter  aided  in  the  derivation  of  an  equation  applied  to  an  unrelated  set  of  gait  data  that 
achieved  shadow  removal.  The  shadow  of  an  individual  can  be  removed  without  overly 
compromising  the  integrity  of  the  subject  walking  across  the  filed. 
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V.  Conclusions  and  Recommendations 


Chapter  Overview 

This  chapter  presents  conclusions  derived  from  results  obtained  in  this  thesis.  In 
addition,  future  of  research  associated  with  the  type  of  algorithm  employed  in  this  thesis 
will  be  suggested. 

Conclusions  of  Research 

This  thesis  presents  a  novel  way  to  produce  the  binary  silhouette  of  an  individual 
free  from  the  effects  of  shading.  The  developed  algorithm  seeks  to  eliminate  shadow 
from  images  prior  to  the  utilization  of  another  algorithm  that  seeks  to  create  a  binary 
silhouette  from  a  subject  moving  across  the  frame. 

It  is  the  contention  of  this  thesis  that  the  methodology  presented  accounts  for 
differently  colored  backgrounds.  The  ellipsoidal  critical  region  employed  presents  an 
attempt  to  remove  pixels  that  are  the  same  color  as  the  average  background  pixel,  though 
darker,  and  neglect  data  that  is  not  related  to  the  color  of  the  background,  which  is  likely 
an  individual  moving  across  the  background.  The  algorithm  allows  for  the  removal  of  a 
large  percentage  of  shaded  colors. 

The  research  question  asked  whether  it  is  possible  to  remove  shadow  from  a  video 
feed  without  compromising  the  integrity  of  objects  to  be  observed.  The  algorithm 
derived  showed  that  it  is  possible  to  remove  shadow  from  a  video  feed  without  losing  a 
large  amount  of  pertinent  data.  Furthermore,  if  small  portions  of  the  data  associated  with 
the  individual  are  misclassilled,  smoothing  and  filling  algorithms  aid  in  the  accurate 
rendering  of  a  binary  silhouette. 
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Recommendations  for  Future  Research 


This  research  is  a  small  step  toward  the  overachieving  goal  of  realizing  gait 
recognition  as  a  feasible  biometric.  There  are  still  avenues  to  be  researched  in  the  realm 
of  shadow  removal  and  the  usage  of  the  binary  silhouettes. 

The  control  was  a  single  subject  walking  perpendicular  to  a  video  camera.  This  is 
a  rather  stringent  setting.  Variables  such  as  plants  or  moving  objects  will  add  an  amount 
of  noise  not  handled  by  the  shadow  removal  and  background  subtraction  algorithms.  The 
addition  of  multiple  subjects  in  the  frame  to  be  tested  would  rest  in  multiple  silhouettes. 
The  multiple  silhouettes  would  need  to  be  manually  separated  by  researchers  to  complete 
an  analysis.  Also,  if  two  subjects  were  walking  together,  the  algorithms  would  create  a 
single  indistinguishable  silhouette. 

In  addition,  if  the  camera  is  not  stationary,  the  use  of  an  average  background 
becomes  meaningless.  There  are  two  cases  of  camera  movement  that  need  further 
research.  The  camera  could  be  oscillating  left  to  right  from  a  fixed  location.  The  camera 
could  be  moving  due  to  outside  influences  like  wind  or  a  person  may  be  holding  the 
camera. 

Furthermore,  the  speed  and  space  needed  by  an  algorithm  is  of  particular 
importance.  The  shadow  removal  and  background  subtraction  algorithms  test  every  pixel 
in  every  frame.  With  video  data  incorporating  higher  pixel  counts,  the  algorithm  will 
take  longer  to  achieve  a  result  and  the  size  of  stored  average  background  will  increase. 
The  test  area  could  be  scaled  down  to  only  include  the  individual  in  a  frame. 
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The  use  of  the  binary  silhouettes  will  allow  researchers  to  extract  usable 
information  concerning  a  subject  being  analyzed.  With  the  proper  algorithms,  a  subject’s 
height,  weight,  gender  could  be  determined. 

There  are  a  multitude  of  covariates  that  the  shadow  removal  and  background 
subtraction  algorithm  do  not  account  for.  For  example,  a  subject  could  carry  a  briefcase 
to  obstruct  the  rendering  of  a  binary  silhouette.  Moreover,  the  wearing  a  dress  or  flowing 
robes  would  obstruct  the  legs  of  a  subject.  Further  research  is  also  needed  to  test  the 
effect  of  subjects  carrying  heavy  loads.  There  are  several  avenues  open  for  further 
research. 

Summary 

This  chapter  discussed  recommendations  for  future  research  and  the  conclusions 
of  the  research.  The  research  shows  that  it  is  possible  to  remove  the  shadow  of  an 
individual  without  losing  data  needed  for  further  processing.  In  addition,  it  has  been 
demonstrated  that  a  binary  silhouette  can  be  rendered  automatically  from  only  the  input 
of  test  frames.  There  is  still  a  multitude  of  research  to  be  accomplished  in  the  field  of 
gait  recognition.  This  thesis  moves  towards  the  ultimate  goal  of  an  automated  gait 
recognition  algorithm. 
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Appendix  A.  MATLAB  Algorithms 


Function:  Load  Video 

function  LoadVideo 

%The  program  loads  video  from  a  camera  into  a  computer 

clear 

clc 

close  all 

%demo ( ' toolbox ' , ' image  processing ' ) 
imaqlnfo  =  imaqhwinfo 

%  contains  information  about  the  image  acquisition  adaptors  available 

on  the  system 

imaqlnfo . InstalledAdaptors 

Winlnfo  =  imaqhwinfo (' winvideo ' ) 

vid  =  videoinput (' winvideo 1 )  ; 

set (vid,  ' FramesPerTrigger ' , 200) 

start (vid) 
wait (vid) 

[f,  t]  =  getdata (vid) ; 
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Function:  Write  Multiple  Images 


function  Writemulti (start, stop, x, y, w, h) 

%Write  multiple  images  to  file 

for  count  =  start: stop, 

red  =  g count) ;  %assignment 

filename  =  ([' red ', num2str (count ),'. jpg ']); %creates  file 
imwrite (  red,  filename  );%writes  file 

end 
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Function:  Crop  Images 

function  [I,rsmall]  =  Crop (start, stop, x, y, w, h) 

%Will  grab  a  specified  range  of  pixels  and  save  them  to  file% 

%start(l)  First  frame  of  no  shadow 

%stop(l)  Last  frame  before  shadow 

%x  coordinate  of  left  of  box 

%y  coordinate  of  top  of  box 

%w  width  of  box 

%h  height  of  box 

for  i=start ( 1 ) : stop ( 1 ) , 

I  =  imread ( [ ' C : \Documents  and  Settings\Owner\Desktop\Thesis\ ' 
sprintf ( ' red%i . jpg ' , i) ] ) ; 
r small  =  imcrop(I,[x  y  w  h] ) ; 

%removes  croppped  section  from  image 

filename  =  ( [ ' rsmall ' , num2str ( i-start (1 ) +1 ) , ' . jpg ' ] ) ;  %creates 
filename  connected  to  each  individual  jpg 

imwrite (rsmall,  filename  );  %writes  cropped  file  to  file 

end 
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Function:  Three-Dimensional  Data  Plots 


function  DataPlots 

%3d  plot  of  pixel  values 

start (1)=  1;  %First  frame  of  no  shadow 
stop(l)  =  34;  %Last  frame  of  no  shadow 

start (2)  =  45;%First  frame  with  shadow 
stop (2)  =  150;%Last  frame 

for  i=start ( 1 )  : stop ( 1 )  , 

I  i-start  ( 1 ) +1 )  =  imread ( [ ' C : \Documents  and 

Settings \Owner\Desktop\Thesis\ '  sprintf ( ' small%i . jpg '  ,  i) ] )  ; 

end 

for  i=start ( 1 ) : stop ( 1 ) , 

f igure=plot3 (mean (mean ( I ( : , : , 1 , i) ) ) , mean (mean (I(:, : , 2 , i ) ) ) , mean (m 
ean (I (:,:, 3 , i %3-d  plot  of  first  vs  last  frame 
hold  on 

end 

for  i=start (2) : stop (2) , 

I(:,:,:,i)  =  imread ([' C : \Documents  and 

Settings \Owner\Desktop\Thesis\ '  sprintf ( ' small%i . jpg ' , i) ] ) ; 

end 

for  i=start (2) : stop (2) , 

f igure=plot3 (mean (mean (I ( : , : , 1 , i ) ) ) , mean (mean ( I ( : , : , 2 , i ) ) ) , mean (m 
ean ( I ( : , : , 3 , i ) ) ) , ' ro ' ) ; %3-d  plot  of  first  vs  last  frame 
hold  on 
grid  on 

axis ( [0  256  0  256  0  256] ) 
xlabel ( ' Red ' ) 
ylabel ( ' Green ' ) 
zlabel ( ' Blue ' ) 

title (' Example  of  blue  background') 

end 

hold  off 
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Function:  Display  Average  RGB  Intensities 


function  [A,photoavg]  =  RGBBackground (start, stop) ; (Samler,  2006:60) 
%  Reads  jpegs  into  matlab,  then  finds  an  avg  RGB  intensity  for 
%  background,  gives%  back  an  average  background  picture.  Also 
%  gives  the  background  picture  in  subplots  of  the  Red,  Green 
%  and  Blue  intensities. 

close  all 

for  i=start ( 1 ) : stop ( 1 ) , 

A (i-start ( 1 ) +1 , :, :, :) =imread ( [ ' C : \ Documents  and 
Settings\Owner\Desktop\Thesis\ '  sprintf ( ' gsmall%i . jpg ' , i) ] ) ; 

end 

s  =  size (A, 4);  %Displays  average  photo  as  image 

photoavg=mean (A) ; 

photoavg  =  squeeze (photoavg) ; 

image (uint8 (photoavg) ) ; 

figure  %displays  3  intensities  1-red  2-green  3-blue 
colormap (gray (256) ) ; 

subplot (3,1,1) ; image (photoavg ( : ,  : ,  1 ) ) 
subplot (3,1,2) ; image (photoavg ( : ,  : ,  2 )  ) 
subplot (3,1,3) ; image (photoavg ( : , : , 3 ) ) 
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Function:  Computation  of  Three-Dimensional  Distances 

function  DistanceValues ; 

%Computes  distances  from  a  point  to  a  3d  vector 

start (1)=  1;  %First  frame  of  no  shadow 
stop(l)  =  34;  %Last  frame  of  no  shadow 

start  (2)  =  55; 
stop (2)  =  150; 

for  i=start ( 1 ) : stop ( 1 ) , 

A  (i-start ( 1 ) +1 )  =  imread ( [ ' C : \Documents  and 
Settings \Owner\Desktop\Thesis\ '  sprintf ( ' rsmall%i . jpg ' , i) ] ) ; 

end 

photoavg  =  (mean (A) ) ;  %Average  photo 

A  =  squeeze (photoavg) ; %Takes  left  column  vector  away 
A=intl 6 (A) ; 

for  i=start (2)  : stop  (2) , 

I(:,:,:,i)  =  imread ([' C : \Documents  and 

Settings \Owner\Desktop\Thesis\ '  sprintf ( ' rsmall%i . jpg ' , i) ] ) ; 

end 

for  n  =  1:200;  %cycle  through 

for  m  =  1:100;  %cycle  through 
for  i=start  (2)  : stop (2) , 

d  =  double (A (m, n, 1 )) ; 
e  =  double  (A  (m,  n,  2  )  )  ; 

f  =  double  (A  (m,  n,  3)  )  ; 

a  =  double  (I  (m,  n,  1,  i)  )  ; 

b  =  double  (I  (m,  n,  2,  i)  )  ; 

c  =  double  (I  (m,  n,  3,  i)  )  ; 

dl (i-start (2) +1) =sqrt ( (a-d) A2  +  (b-e) A2  +  (c- 
f ) A2) *sin (acos ( ( (a-d) *d  +  (b-e) *e+ (c-f ) *f ) / (sqrt ( (a-d) A2  + 
(b-e) A2  +  (c-f ) A2 ) *sqrt (dA2+eA2+f A2 ) ) ) ) ; 

end 

end 

end 

avg  =  mean(dl); 
s  =  std(dl) 
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Function:  Computation  of  Projected  Three-Dimensional  Distances 

function  Distance Pro j ectedValues ; 

%Computes  a  projected  shadow  line 

%Computes  the  3d  distance  of  point  to  shadow  line 
start (1)=  1;  %First  frame  of  no  shadow 
stop(l)  =  34;  %Last  frame  of  no  shadow 

start (2)  =  116; 
stop (2)  =  150; 

for  i=start ( 1 ) : stop ( 1 ) , 

P  i-start ( 1 ) +1 )  =  imread ( [ ' C : \Documents  and 

Settings \Owner\Desktop\Thesis\ '  sprintf ( ' small%i . jpg ' , i) ] ) ; 

end 

for  i=start ( 1 )  : stop ( 1 )  , 

Vl= [ [mean (mean ( P ( : , : , 1 , i ) ) ) ; 0 ] ] ; %average  color  of  background 
V2= [mean (mean ( P ( : ,  : ,  2 ,  i )  )  )  ;  0  ]  ; 

A=plot (VI , V2 %plot  average  color  pixel  to  origin 
hold  on 

S  (  :  ,  i)  =;[V1  (1)  ;  V2  (1)  ]  ; 

ml ( i ) = (V2 ( 1 ) /VI ( 1 ) ) ; %slope  of  line 

end 

for  i=start (2) : stop (2) , 

I i-start  (2 ) +1 )  =  imread ([' C : \Documents  and 
Settings \Owner\Desktop\Thesis\ '  sprintf ( ' small%i . jpg ' , i) ] ) ; 

end 

for  i=l : (stop (2) -start (2) ) , 

V3= [mean (mean ( I ( : , : , 1 , i ) ) ) ] ; %average  color  of  shadow 
V4= [mean (mean ( I ( : ,  :  ,  2  ,  i )  )  )  ]  ; 

m2= (V4-S (2, i) ) / (V3-S (1, i) ) ; %slope  of  line  from  average  pixel  no 
shadow  vs  shadow 

tatheta ( i) =- (m2-ml (i) ) / (1+ml (i) *m2 ) ; %tangent  of  angle  between  lines 
b=V4-V3* (m2 ) ; %y  intercept 
xo=b/ (-m2) ; %x  intercept 

opp (i) =  tatheta (i) *sqrt ( (S (1, i) /  2)  A2  +  (S ( 2 , i ) / 2)  A2) ; %di stance 
between  line  at  midpoint 

B=plot (V3 , V4 , ' ro ' ) ; %plot  of  shadow  point 

C=plot ( [S ( 1 , i ) ; xo] ,  [S  (2 , i ) ; 0 ] ,  ' -r ' ) ; %plot  of  shadow  line 
legend (  [  A  ,  B  ,  C  ]  ,  'Line  Average  Pixels  to  Origin',  'Pixel 

Values  of  Shadow',  'Line  for  Average  Pixels  to  Shadow'  ); 
grid  on 

axis ( [0  256  0  256] ) 
xlabel ( ' Red ' ) 
ylabel ( ' Green ' ) 

title (' Example  of  Blue  Background') 

end 

hold  off 

q=mean (opp) ; %mean  distance 
r=std (opp) ; %std  distance 

ci=abs (q) +1 . 96*sqrt (abs (r/q) ) ; %upper  confidence  interval 
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Function:  Shadow  Removal  Algorithm 

function  ShadowTest; 

%Test  to  remove  shadow  from  test  images 
close  all 
start ( 1 ) =1 ; 
stop  (1) =36; 


for  i=start ( 1 )  : stop ( 1 )  , 

A ( i-start ( 1 ) +1 )  =  imread ( [ ' C : \Documents  and 
Settings\Owner\Desktop\Thesis\ '  sprintf ( ' rsmall%i . jpg ' , i) ] ) ; 

end 


start=55 ; 
stop=60 ; 

for  i=start : stop, 

P , i-start+1 ) =imread ( [ ' C : \Documents  and 

Settings\Owner\Desktop\Thesis\ '  sprintf ( ' rsmall%i . jpg ' , i) ] ) ; 

%Image  is  stored  as  a  three-dimensional  (m-by-n-by-3)  array  of  integers 

in  the  range  [0,  255]  (uint8) 

end 

photoavg  =  (mean (A) ) ;  %average  picture 

photoavg  =  double (( squeeze (photoavg) )) ;  %changes  4d  to  3d 

tt=  zeros (1,  (stop-start+1) ) ;  %Preload  Variable 

for  m  =  l:100;%cycle  through  720 

for  n  =1:200  ;  %cycle  through  480 

c  =  [ (photoavg (m, n, 1) ); (photoavg (m, n, 2) ); (photoavg (m, n, 3) )] ; 
%Ellipse  end  point 

zl=c ( 1 ) /sqrt (c ( 1 ) A2+c (2 ) A2 ) ; %red  and  green 

z2=c (2 ) /sqrt (c ( 1 ) A2+c (2 ) A2 ) ; %cos  angle  red  and  blue  shift 

Rz=[[zl  -z2  0]',[z2  zl  0] ' , [0  0  1] ' ] ;  % rot at ion  matrix  about  z 

for  i=start : stop, 

d  =  double (P (m, n, 1, i-start+1) ) ; 
e  =  double (P (m, n, 2, i-start+1) ) ; 
f  =  double (P (m, n, 3, i-start+1) ) ; 

g  =  [0 . 0951; 0 . 0486] ; %Coef f icients  for  green  red 
h  =  [0 . 0792; 0 . 0955] ; %coeff icients  for  blue  red 
gp=[d,e]*g;  %output  for  ellipsoid  width 
bp=[e,f]*h;  %output  for  ellipsoid  height 
ah (m,n, i-start+1) =gp; 
bh (m, n, i-start+1 ) =bp; 

points  =  [d;e;f];  %Points  in  ellipse 

cpoints  =  ( (c+ [ 0 ; 0 ; bp/2 ] ) /2 ) -  points ; %Center  ellipse 
g= [ [0; 0; 0] ,  [c+ [-c (1) +sqrt (c(l) A2+c(2) A2+c(3) A2) ;-c  (2) ;- 

c (3) ]  ]  ] ; 


g=Rz'*g;  %transform  of  line  around  z 

y=[l;c (3) /sqrt (c ( 1 ) A2+c  (2 ) A2 ) ]  ; 

%cos  =1  sin  of  angle  between  shifted  line  and 
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Ry=[[y(l)  0  -y (2) ] ' , [0  1  0]',[y(2)  0  y ( 1 ) ] ' ] ; 

%rotation  matrix  about  y 

R  =  Ry*Rz;  %complete  rotation  matrix 

j=sqrt (c (1) A2  +  c(2)A2  +  c ( 3 ) A2+l ) ; %ma j or  axis  of  ellipsiod 
if  gp  >  0; 

if  bp  >  0; 

V=[ [1/ (j/1.9) A2  0  0]',[0  1/ (gp) A2  0]',[0  0 
1 / (bp) A2 ] ' ] ; %Length  axes 

end 

end 

ell=sum( (R*cpoints) .* (V*R*cpoints ) ) ; 

%Matrix  ellipse  equation 

ch (m, n, i-start+1) =ell;  %Matrix  ellipse  equation 
if  ell  <  1, 

tt ( 1 , i-start+1 ) =tt ( 1 , i-start+1 ) +1 ; 
for  p-l:3 

P(m,n,p,i-start+l)= (photoavg (m, n, p) ) ; 

%Replaces  a  pixel  in  an  image  with  the  average  for 
that  pixel 

end 

end 

end 

end 

end 
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Function:  Computation  of  Upper  Confidence  Intervals 

function  Conf id; 

%Compute  confidence  intervals  of 
%distances  of  shadow  line  to  a  point 
clc 
clear 

start (1)=  1;  %First  frame  of  no  shadow 
stop(l)  =  34;  %Last  frame  of  no  shadow 

start (2)  =  116; 
stop (2)  =  150; 

for  i=start ( 1 ) : stop ( 1 ) , 

P ( i-start ( 1 ) +1 )  =  imread ( [ ' C : \Documents  and 
Settings\Owner\Desktop\Thesis\ '  sprintf ( ' small%i . jpg ' , i) ] ) ; 

end 

photoavg  =  (mean (P) ) ;  %average  picture 

photoavg  =  double (( squeeze (photoavg) )) ;  %changes  4d  to  3d 
for  i=start (2) : stop (2) , 

I i-start (2 ) +1 )  =  imread ([' C : \Documents  and 
Settings\Owner\Desktop\Thesis\ '  sprintf ( ' small%i . jpg ' , i) ] ) ; 

end 

for  i=l : (stop (2) -start (2) ) , 

for  m  =  l:100;%cycle  through  720 

for  n  =  1:200;  %cycle  through  480 

V  =  [photoavg (m, n, 1) , (photoavg (m, n, 2) ) , (photoavg (m, n, 3) ) ] ; 
%Ellipse  end  point 

ml  =  (V (3) /V (1) ) ; %slope  of  line 

VI  =  [double (I (m, n, 1, i) )]; %average  color  of  shadow 
V2  =  [double  (I  (m,  n,  2,  i)  )  ]  ; 

V3  =  [double  (I  (m,  n,  3,  i)  )  ]  ; 
m2= (V ( 3 ) -V3 ) / ( V ( 1 ) -VI ) ; 

%slope  of  line  from  average  pixel  no  shadow  vs  shadow 
tatheta= (m2-ml) / (l+ml*m2) ; %tangent  of  angle  between  lines 

opp (m, n, i) =  -tatheta*sqrt ( ( (V(l) ) / 2)  A2+ ( (V(2) ) / 2)  A2) ; 
%distance  between  line  at  midpoint 

end 

end 

end 

q=squeeze (mean (squeeze (mean (opp) ) ) ) ; %mean  distance 

r=std (q) ; %std  distance 

for  i=l : (stop (2) -start (2) ) , 

ci (i) =abs (q (i) ) +1 . 96*sqrt (abs (r*l/q (i) ) ) ; %upper  confidence  interval 

end 

max (ci) 
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Function:  Computation  of  Coefficients  for  Shadow  Removal  Algorithm 


function  Coef f ; 

%Compute  coefficients  used  in  Test.m 

close  all 
start ( 1 ) =1 ; 
stop (1) =36; 


for  i=start ( 1 ) : stop ( 1 ) , 

A ( i-start ( 1 ) +1 )  =  imread ( [ ' C : \Documents  and 
Settings\Owner\Desktop\Thesis\ '  sprintf ( ' small%i . jpg ' , i) ] ) ; 

end 

photoavgB  =  (mean (A) ) ;  %average  picture 

photoavgB  =  double (( squeeze (photoavgB) )) ;  %changes  4d  to  3d 
for  i=start ( 1 ) : stop ( 1 ) , 

A ( i-start ( 1 ) +1 )  =  imread ([' C : \Documents  and 
Settings \Owner\Desktop\Thesis\ '  sprintf ( ' rsmall%i . jpg ' , i) ] ) ; 

end 

photoavgR  =  (mean (A) ) ;  %average  picture 

photoavgR  =  double (( squeeze (photoavgR) )) ;  %changes  4d  to  3d 
for  i=start ( 1 ) : stop ( 1 ) , 

A ( i-start ( 1 ) +1 )  =  imread ([' C : \Documents  and 
Settings\Owner\Desktop\Thesis\ '  sprintf ( ' gsmall%i . jpg ' , i) ] ) ; 

end 

photoavgG  =  (mean (A) ) ;  %average  picture 

photoavgG  =  double (( squeeze (photoavgG) )) ;  %changes  4d  to  3d 

T ( 1 , : ) = [mean (mean (photoavgG ( : , : , 1 ) ) ) , mean (mean (photoavgG ( : , : , 2 ) ) ) , mean ( 
mean (photoavgG ( : , : , 3 ) ) ) ] ; 

T (2 , : ) = [mean (mean (photoavgR ( : , : , 1 ) ) ) , mean (mean (photoavgR ( : , ; , 2 ) ) ) , mean ( 
mean (photoavgR ( : ,  : , 3 ) ) ) ]  ; 

T ( 3 ,  : )  =  [mean (mean (photoavgB ( : ,  : , 1 ) ) ) , mean (mean (photoavgB ( : ,  ; , 2 ) ) ) , mean ( 
mean (photoavgB ( : ,  : ,  3) ) )  ]  ; 

RB=[T(:,1)  ,T(:,3)  ]  ; 

RG= [ T ( : , 1 ) , T  (  :  , 2 )  ]  ; 

RB2  =  [43.2667;  21.3211;  13.2447]; 

RG2  =  [11.3852;  2.2839;  3.1505]; 

coefg=RB\RB2 

coefB=RG\RG2 
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Function:  Plot  of  Rotation  of  Ellipsoid 

function  RotationPlot; 

%Displays  the  order  and  placement  of  a  line  after  rotations 

clear 

n=500000 ; 

c  =  [91; 61; 107];  %Ellipse  end  point 

points  =  rand ( [3, n] ) *100;  %Points  in  ellipse 

cpoints  =  points  -  repmat ( (c+ [0; 0; 0] ) /2, [1  n]);%Center  ellipse 

zl=c ( 1 ) /sqrt (c ( 1 ) A2+c (2 ) A2 ) ; %red  and  green 
z2=c (2 ) / sqrt (c(l)A2+c(2)A2); 

Rz=[[zl  -z2  0] ' , [z2  zl  0 ] ' , [ 0  0  1]'];  % rot at ion  matrix  about  z 
g= [  [0; 0; 0] ,  [c+ [-C (1) +sqrt (c (1 ) A2+c (2 ) A2+c (3 ) A2 ) ; -c (2 ) ; -c  (3 ) ] ] ] ; 

%Line  on  xy  plane 

A=plot3  (  [g  (1,  :)],  [g  (2,  :)],  [g  (3,  :)],  '  o-g'  )  ; 

%Plot  Original  Line 
hold  on 

g=Rz'*g;  %Transf ormation  of  line  about  z 
%B=plot3 ( [g  (1,  : ) ]  ,  [g  (2,  : ) ]  ,  [g  (3,  : ) ]  ,  ' o-b ' ) ; 

%Line  after  transformation 

y=  [1;  c  (3)  / sqrt  (c  ( 1 )  A2+c  (2  )  A2  )  ]  ; 

%Angle  for  y  transformation  leaving  x  alone 

Ry=[[y(l)  o  -y (2) ] ' , [0  1  0]',[y(2)  0  y ( 1 ) ] ' ] ; 

%rotation  matrix  about  y 

R  =  Ry*Rz;  %full  rotation  matrix 

C=plot3 ( [g ( 1 , :)], [g(2, :)], [g(3, :)], ' o-r ' ) ; 

%Line  Trying  to  get 

l=sqrt (c (1) A2+c (2) A2+c (3) A2) +2; %major  axis  of  ellipsiod 

V=[ [1/ ( (1+25) /2) A2  0  0]',[0  1/ (10) A2  0 ]  '  ,  [  0  0  1/ (18) A2]  ' ] ; %Length  axes 

ell=sum( (R*cpoints) .* (V*R*cpoints ) ) ; 

%Matrix  ellipse  equation 
index  =  find(ell<2); 

D=plot3 (points ( 1 , index) , points (2 , index) , points ( 3 , index) ,'.'); 

%3-d  plot  of  first  vs  last  frame 

legend (  [A  ,C,D]  ,  'Original  Line', 'First  Rotation'  , 'Ellipsoid  Test 

Region '  ) ; 

grid  on 

axis  (  [0  100  0  70  0  120] ) 
xlabel ( ' Red ' ) 
ylabel ( ' Green ' ) 
zlabel ( ' Blue ' ) 

title (' Transformations  of  Test  Area  Ellipsoid') 

plot3 (  [ 0  c ( 1 )  ] ,  [ 0  c  (2 ) ] ,  [ 0  c  (3 ) ] ,  '  . -r ' ) ;  %Line  Trying  to  get  to 


hold  off 
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Function:  Shadow  Removal  and  Back  Ground  Subtraction 


function  Test; 

%Removes  shadow  then  runs  background  subtraction 

close  all 

start ( 1 ) =1 ; 

stop ( 1 ) =5 ; 

start (2 ) =  124 ; 

stop (2) =  128; 

for  i=start ( 1 ) : stop ( 1 ) , 

A(i-start(l)+l, : , :) =imread ( [ ' C : \ Documents  and 

Settings\Owner\Desktop\Thesis\Pass#l\ '  sprintf ( ' frame%i . jpeg ' , i) ] ) 

end 

for  i=start (2) : stop (2) , 

A ( stop ( 1 ) +i- start (2 ) +1 , : , : , :) =imread ( [ ' C : \ Documents  and 
Settings\Owner\Desktop\Thesis\Pass#l\ '  sprintf ( ' frame%i . jpeg ' , i) ] ) 

end 


start=55 ; 
stop=57 ; 

for  i=start : stop, 

P , i-start+1 ) =imread ( [ ' C : \Documents  and 

Settings\Owner\Desktop\Thesis\Pass#l\ '  sprintf ( ' frame%i . jpeg ' , i) ] ) 
%Image  is  stored  as  a  three-dimensional  (m-by-n-by-3)  array  of 
integers  in  the  range  [0,  255]  (uint8) 

end 

photoavg  =  (mean (A) ) ;  %average  picture 

photoavg  =  double (( squeeze (photoavg) )) ;  %changes  4d  to  3d 


for  m  =  l:480;%cycle  through  720 

for  n  =1:720  ;  %cycle  through  480 

c  =  [ (photoavg (m, n, 1) );  (photoavg (m,  n,  2) );  (photoavg (m,  n,  3) )]  ; 

%Ellipse  end  point 

zl=c ( 1 ) /sqrt (c ( 1 ) A2+c (2 ) A2 ) ; %red  and  green 

z2=c (2 ) /sqrt (c ( 1 ) A2+c (2 ) A2 ) ; %cos  angle  red  and  blue  shift 

Rz=[[zl  -z2  0] ' , [z2  zl  0] ' , [0  0  1] ' ] ;  % rot at ion  matrix  about 

for  i=start : stop, 

d  =  double (P (m, n, 1, i-start+1) ) ; 
e  =  double (P (m, n, 2, i-start+1) ) ; 
f  =  double (P (m, n, 3, i-start+1) ) ; 

q  =  [0 . 0783; 0 . 0512] ; %coef f icients  for  green  red 
h  =  [ 0 . 24 67 ; 0 . 0584 ] ; %Coeff icients  for  blue  red 
p  =  [0 . 1294; 0 . 0242] ; %Coeff icients  for  blue  green 
gp=[d,e]*q;  %output  for  ellipsoid  width 
bpl=[d,f]*h;  %output  for  ellipsoid  height 
bp2=[e,f]*p;  %output  for  ellipsoid  height 
bp=max (bpl ,  bp2 ) ; 
ah (m, n, i-start+1 ) =gp; 
bh (m, n, i-start+1 ) =bp; 
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points  =  [d;e;f];  %Points  in  ellipse 

cpoints  =  ( (c+ [ 0 ; 0 ; bp/2 ] ) /2 ) -  points ; %Center  ellipse 
g= [ [0; 0; 0] ,  [c+ [-c (1) +sqrt (c(l) A2+c(2) A2+c(3) A2) ;-c (2)  ;  - 
c (3) ]  ]  ]  ; 

g=Rz'*g;  %transform  of  line  around  z 

y= [ 1 ; c ( 3 ) /sqrt (c ( 1 ) A2+c (2 ) A2 ) ] ; %cos  =1  sin  of  angle  between 
shifted  line  and 

Ry= [ [ y ( 1 )  o  -y (2) ] ' , [0  1  0 ] ' , [y (2 )  0  y(l)]'];  %rotation 
matrix  about  y 

R  =  Ry*Rz;  %complete  rotation  matrix 

j=sqrt (c (1) A2  +  c(2)A2  +  c ( 3 ) A2+l ) ; %ma j or  axis  of  ellipsiod 
if  gp  >  0; 

if  bp  >  0; 

V=[ [1/ (j/1.9) A2  0  0]',[0  1 / (gp/2 ) A2  0]',[0  0 
1 /  (bp/2 ) A2 ]  ' ] ; %Length  axes 

end 

end 

ell=sum( (R*cpoints) .* (V*R*cpoints ) ) ; 

%Matrix  ellipse  equation 

ch (m, n, i-start+1) =ell;  %Matrix  ellipse  equation 
if  ell  <  1, 

for  p=l : 3 

P(m,n,p,i-start+l)= (photoavg (m, n, p) ) ; %Replaces  a 
pixel  in  an  image  with  the  average  for  that  pixel 

end 

end 

end 

end 

end 

k  =  5 . 2 ; %Standard  Deviations ( Samler,  2006:60) 
for  i=l : size (A, 1) ; 

dif favgl ( i )  =  squeeze ( inti 6 (A ( i )  -  inti 6 (photoavg) ; 
%Used  to  help  find  variances  of  differences 

end 

Asize  =  prod (size (diffavgl 1) ))  ; 
for  i=l:3 

v (i) =  var (reshape (diffavgl (:, :,i), Asize, 1,1,1)); 

end 

for  i=l : size (P, 4) ; 

I2=intl6 (photoavg) -intl6  (P  ( : ,  : ,  : , i) ) ; 

I  ( : ,  :,  : , i)  =  (12 ( : ,  :,1)  . A2 / ( k*v ( 1 ) ) +12 ( : ,  :,2)  . A2/ (k*v (2) ) +12 ( : ,  :,3)  ,A2/(k 
*v ( 3 ) ) >1 ) ; 

I  (:,:,:, i)  =  medfilt2 (I(:,:,:,i),[20  20]); 

I(:,:,:,i)  =  imf ill (( I (:,:,:, 1 )),' holes ') ; 

end 

figure 

for  i=l : size (P, 4) ; 

subplot (1, 3, i) % (rows, columns, frames) 
imshow ( I ( : , : , : , i ) ) 

end 
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